Repository navigation
feat(catalogue): run LongMemEval on one question type - #32
Merged
Merged
Conversation
A question_type option keeps one type's questions, abstention ones included, in dataset order and before --limit. The default, all, runs every question as before.
🟢 PR judge · 7 of 7 yes · #32 ↔ #15
Acceptance criteria, 2 of 2 met
Checks the script runs itself, 4 of 4 pass
|
aitianshi
changed the title
aitianshi
added a commit
that referenced
this pull request
Nov 10, 2026
## What Pull request titles become conventional commit subjects, `type(scope): summary`. The repository squash-merges with the PR title as the commit subject, so the title is what `main`'s history shows. AGENTS.md states the rule with its types (`feat`, `fix`, `perf`, `refactor`, `docs`, `test`, `ci`, `build`, `chore`, `revert`) and scopes (the parts of Nullius), and the `/file-issue` skill, which assistants use to open PRs, applies it. No check enforces it. ## Touches docs and site · CI and skills ## Verified - Backend suite `python3 app/tests/parallel.py`, OK. - The open PRs #29, #30, #32 and #34 already use the form. ## Risk Documentation only; earlier commit subjects on `main` are not rewritten. Undone by a revert.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15
What
LongMemEval gets a
question_typeoption:all(the default, every question as today) or one of the dataset's six types (knowledge-update,temporal-reasoning,multi-session,single-session-user,single-session-assistant,single-session-preference). The driver'sunits()keeps only that type's questions, abstention ones included (theirquestion_typeis the base type's), in dataset order; the runner applies--shardand--limitafterunits(), so the filter comes first. The option applies to every setup (Original, the model setup and any memory). The type is read while the line index is built, so no extra pass over the file. The settings inventory names it, the benchmark's docs page is regenerated, and the test pinning LongMemEval's default options now includes it.Touches
catalogue · docs and site
Verified
app/tests/test_longmemeval_setups.py(against the suite's fake upstream): one type keeps that type's questions with its abstention variant, in order; no option andallkeep every question; the option resolves for all three setups.longmemeval_oracle.json(pinned sha256 matched), through the driver'sunits:knowledge-update78 (6 abstention),all500; the other types 133, 133, 70, 56, 30.python3 app/tests/parallel.pyand route suiteapp/ui_tests, OK.Risk
A recipe that names no question type gets
all, today's behaviour. Each dataset entry'sunits: 500still counts the whole file, so an estimate shown before a filtered run can overstate its size. Undone by a revert.